Add jvm aware setting and max num docs settings for batching docs for percolate queries by eirsep · Pull Request #1435 · opensearch-project/alerting

eirsep · 2024-02-19T10:19:09Z

With these changes the number of docs submitted in a single percolate query is not naively set per shard or per index. Rather we have 2 settings to decide how many docs to submit for percolate query in doc level monitor

Solves the followping problems

IllegalStateException[Failed to run percolate search for sourceIndex [cloudtrail_alias] and queryIndex [.opensearch-sap-cloudtrail-detectors-queries-000001] for 180000 document(s)];
 nested: SearchPhaseExecutionException[SearchTask was cancelled]; 
nested: TaskCancelledException[org.opensearch.core.concurrency.OpenSearchRejectedExecutionException: cancelled task with reason: Cancellation timeout of 5m is expired]; 
nested: OpenSearchRejectedExecutionException[cancelled task with reason: Cancellation timeout of 5m is expired];

"error_message" : "IllegalStateException[Failed to run percolate search for sourceIndex [log-aws-cloudtrail-2023-08] and queryIndex [.opensearch-sap-cloudtrail-detectors-queries-000001] for 10000 document(s)]; 
nested: SearchPhaseExecutionException[all shards failed]; 
nested: [cancelled task with reason: heap usage exceeded [45.9mb >= 9.2mb]]; 
nested: OpenSearchRejectedExecutionException[cancelled task with reason: heap usage exceeded [45.9mb >= 9.2mb]];

Issue #, if available:
Optimize doc level monitor performance: Batch docs for percolate query searches based on available memory and cpu #1353

Description of changes:

Log message from opensearch cluster, when setting is at 40k docs per batch and 10% of heap to break batch and perform percolate query for ingestion rate of 250K docs per minute

Monitor org.opensearch.client.node.NodeClient@1440ce1 PERF_DEBUG: Percolate query time taken millis = 9.4s

Old Latency of percolate query : 5+ minutes leading to cancellation. New latency <1 minute

…ueries Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

alerting/src/main/kotlin/org/opensearch/alerting/DocumentLevelMonitorRunner.kt

alerting/src/main/kotlin/org/opensearch/alerting/AlertingPlugin.kt

alerting/src/main/kotlin/org/opensearch/alerting/DocumentLevelMonitorRunner.kt

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

alerting/src/main/kotlin/org/opensearch/alerting/MonitorRunnerService.kt

alerting/src/main/kotlin/org/opensearch/alerting/settings/AlertingSettings.kt

engechas · 2024-02-19T21:51:53Z

alerting/src/main/kotlin/org/opensearch/alerting/DocumentLevelMonitorRunner.kt

+        val queryIndices =
+            docs.map { monitorMetadata.sourceToQueryIndexMapping[it.second.indexName + monitor.id] }.distinct()


Is there a case where there would be more than one query index here? If so, should only docs with query index A be submitted to a percolator query against query index A? Currently it seems like we are sending all docs to a percolate query against both index A and index B

Great question.. will consult @sbcd90 for advice here

Currently for a single monitor there will be only one query index acc. to @sbcd90

Got it - I could see this get overlooked if that changes in the future. Would it be too difficult to break up the execution of the percolate query to be per-query-index? It shouldn't have any performance/functionality impacts currently with only one query index, but would guard against that changing down the line.

correctness would be intact so that's ok. we would anyway evaluate performance if we make such a change of having to query multiple indices

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

opensearch-trigger-bot · 2024-02-22T22:27:31Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/alerting/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/alerting/backport-2.x
# Create a new branch
git switch --create backport-1435-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 f643454a22b239a283e47c29222e561d238de42e
# Push it to GitHub
git push --set-upstream origin backport-1435-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/alerting/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport-1435-to-2.x.

… percolate queries (opensearch-project#1435) * add jvm aware and max docs settings for batching docs for percolate queries Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix stats logging Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add queryfieldnames field in findings mapping Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* Add jvm aware setting and max num docs settings for batching docs for percolate queries (#1435) * add jvm aware and max docs settings for batching docs for percolate queries Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix stats logging Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add queryfieldnames field in findings mapping Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * optimize to fetch only fields relevant to doc level queries in doc level monitor instead of entire _source for each doc (#1441) * optimize to fetch only fields relevant to doc level queries in doc level monitor Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix test for settings check Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix ktlint Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * clean up doc level queries on dry run (#1430) Signed-off-by: Joanne Wang <jowg@amazon.com> * optimize sequence number calculation and reduce search requests in doc level monitor execution (#1445) * optimize sequence number calculation and reduce search requests by n where n is number of shards being queried in the executino Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix tests Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * optimize check indices and execute to query only write index of aliases and datastreams during monitor creation Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix test Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add javadoc Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add tests to verify seq_no calculation Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> Signed-off-by: Joanne Wang <jowg@amazon.com> Co-authored-by: Joanne Wang <jowg@amazon.com>

…ject#1307) * Added 2.11.1 release notes. * Added 2.11.1 release notes. --------- (cherry picked from commit 06c1b8a) Signed-off-by: AWSHurneyt <hurneyt@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> fix workflow security tests in alerting (opensearch-project#1310) (opensearch-project#1311) Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> Increment version to 2.12.0-SNAPSHOT (opensearch-project#1239) Signed-off-by: opensearch-ci-bot <opensearch-infra@amazon.com> Co-authored-by: opensearch-ci-bot <opensearch-infra@amazon.com> [Backport 2.x] Reference get monitor and search monitor action / request / responses from common-utils (opensearch-project#1315) * Use get monitor action / req / resp from common-utils Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com> * Dummy commit to retrigger Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com> --------- Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com> optimize doc-level monitor execution workflow for datastreams (opensearch-project#1302) (opensearch-project#1322) Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> Update to Gradle 8.5 (opensearch-project#1369) (opensearch-project#1371) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> [Backport 2.x] Inject namedWriteableRegistry during ser/deser of SearchMonitorAction (opensearch-project#1382) (opensearch-project#1384) * Inject namedWriteableRegistry during ser/deser of SearchMonitorAction (opensearch-project#1382) Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com> * remove bin files Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com> * remove core bin Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com> --------- Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com> Don't attempt to parse workflow if it doesn't exist (opensearch-project#1346) (opensearch-project#1359) (cherry picked from commit 733fd4e) Signed-off-by: Chase Engelbrecht <engechas@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Set docData to empty string if actual is null (opensearch-project#1325) (opensearch-project#1334) (cherry picked from commit 008e076) Signed-off-by: Chase Engelbrecht <engechas@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> removed default admin credentials for alerting (opensearch-project#1399) (opensearch-project#1400) (cherry picked from commit 3c50f7d) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Dennis Toepker <toepkerd@amazon.com> ipaddress lib upgrade as part of cve fix (opensearch-project#1397) (opensearch-project#1407) (cherry picked from commit 8d59060) Signed-off-by: Riya Saxena <riysaxen@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Bulk index findings and sequentially invoke auto-correlations (opensearch-project#1355) (opensearch-project#1410) * Bulk index findings and sequentially invoke auto-correlations * Bulk index findings in batches of 10000 and make it configurable * Addressing review comments * Add integ tests to test bulk index findings * Fix ktlint formatting --------- (cherry picked from commit b561965) Signed-off-by: Megha Goyal <goyamegh@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Add 2.12 release notes (opensearch-project#1408) (opensearch-project#1413) * Add 2.12 release notes * Fix release notes PR * Add 2 more PRs --------- (cherry picked from commit b10eaad) Signed-off-by: Chase Engelbrecht <engechas@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> [Backport 2.x] Implemented cross-cluster monitor support (opensearch-project#1404) (opensearch-project#1412) * Implemented cross-cluster monitor support (opensearch-project#1404) * Updated alert mappings to accommodate cross-cluster cluster metrics monitors. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Implemented support for cross-cluster cluster metrics monitors. Implemented GetRemoteIndexes API to populate the frontend UI with details regarding the remote clusters, and indexes. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Fixed a writeable test after changing QueryLevelTriggerRunResult from a data class to an open class for inheritability. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Fixed ktlint errors. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Removed changes to IndexUtils as they're only needed by doc monitors. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> --------- Signed-off-by: AWSHurneyt <hurneyt@amazon.com> (cherry picked from commit ea36996) Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Fixed a test. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> --------- Signed-off-by: AWSHurneyt <hurneyt@amazon.com> Add publishToMavenLocal in build.sh (opensearch-project#1418) (opensearch-project#1419) (cherry picked from commit 4cdc1d1) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> fix for MapperException[the [enabled] parameter can't be updated for the object mapping [metadata.source_to_query_index_mapping] (opensearch-project#1432) (opensearch-project#1434) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> bacport PRs opensearch-project#1445, opensearch-project#1430, opensearch-project#1441, opensearch-project#1435 to 2.x (opensearch-project#1452) * Add jvm aware setting and max num docs settings for batching docs for percolate queries (opensearch-project#1435) * add jvm aware and max docs settings for batching docs for percolate queries Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix stats logging Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add queryfieldnames field in findings mapping Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * optimize to fetch only fields relevant to doc level queries in doc level monitor instead of entire _source for each doc (opensearch-project#1441) * optimize to fetch only fields relevant to doc level queries in doc level monitor Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix test for settings check Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix ktlint Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * clean up doc level queries on dry run (opensearch-project#1430) Signed-off-by: Joanne Wang <jowg@amazon.com> * optimize sequence number calculation and reduce search requests in doc level monitor execution (opensearch-project#1445) * optimize sequence number calculation and reduce search requests by n where n is number of shards being queried in the executino Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix tests Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * optimize check indices and execute to query only write index of aliases and datastreams during monitor creation Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix test Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add javadoc Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add tests to verify seq_no calculation Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> Signed-off-by: Joanne Wang <jowg@amazon.com> Co-authored-by: Joanne Wang <jowg@amazon.com> [Backport 2.x] Add an _exists_ check to document level monitor queries (opensearch-project#1425) (opensearch-project#1456) * Add an _exists_ check to document level monitor queries (opensearch-project#1425) * clean up and add integ tests Signed-off-by: Joanne Wang <jowg@amazon.com> * refactored out common method and renamed test Signed-off-by: Joanne Wang <jowg@amazon.com> * remove _exists_ flag Signed-off-by: Joanne Wang <jowg@amazon.com> --------- Signed-off-by: Joanne Wang <jowg@amazon.com> * fix integ test Signed-off-by: Joanne Wang <jowg@amazon.com> --------- Signed-off-by: Joanne Wang <jowg@amazon.com> add distributed locking to jobs in alerting (opensearch-project#1403) (opensearch-project#1458) Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

… percolate queries (opensearch-project#1435) * add jvm aware and max docs settings for batching docs for percolate queries Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix stats logging Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add queryfieldnames field in findings mapping Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

… percolate queries (#1435) * add jvm aware and max docs settings for batching docs for percolate queries Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix stats logging Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add queryfieldnames field in findings mapping Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

… percolate queries (opensearch-project#1435) * add jvm aware and max docs settings for batching docs for percolate queries Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix stats logging Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add queryfieldnames field in findings mapping Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

…#1471) * Bulk index findings and sequentially invoke auto-correlations (#1355) * Bulk index findings and sequentially invoke auto-correlations Signed-off-by: Megha Goyal <goyamegh@amazon.com> * Bulk index findings in batches of 10000 and make it configurable Signed-off-by: Megha Goyal <goyamegh@amazon.com> * Addressing review comments Signed-off-by: Megha Goyal <goyamegh@amazon.com> * Add integ tests to test bulk index findings Signed-off-by: Megha Goyal <goyamegh@amazon.com> * Fix ktlint formatting Signed-off-by: Megha Goyal <goyamegh@amazon.com> --------- Signed-off-by: Megha Goyal <goyamegh@amazon.com> * Add jvm aware setting and max num docs settings for batching docs for percolate queries (#1435) * add jvm aware and max docs settings for batching docs for percolate queries Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix stats logging Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add queryfieldnames field in findings mapping Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * optimize to fetch only fields relevant to doc level queries in doc level monitor instead of entire _source for each doc (#1441) * optimize to fetch only fields relevant to doc level queries in doc level monitor Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix test for settings check Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix ktlint Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix integTests Signed-off-by: Joanne Wang <jowg@amazon.com> * clean up doc level queries on dry run (#1430) Signed-off-by: Joanne Wang <jowg@amazon.com> * optimize sequence number calculation and reduce search requests in doc level monitor execution (#1445) * optimize sequence number calculation and reduce search requests by n where n is number of shards being queried in the executino Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix tests Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * optimize check indices and execute to query only write index of aliases and datastreams during monitor creation Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix test Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add javadoc Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add tests to verify seq_no calculation Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix integ tests again Signed-off-by: Joanne Wang <jowg@amazon.com> --------- Signed-off-by: Megha Goyal <goyamegh@amazon.com> Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> Signed-off-by: Joanne Wang <jowg@amazon.com> Co-authored-by: Megha Goyal <56077967+goyamegh@users.noreply.github.com> Co-authored-by: Surya Sashank Nistala <snistala@amazon.com>

… percolate queries (opensearch-project#1435) * add jvm aware and max docs settings for batching docs for percolate queries Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix stats logging Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add queryfieldnames field in findings mapping Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* use notiifcation snapshot for integ test (#822) Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * Fix build Signed-off-by: Chase Engelbrecht <engechas@amazon.com> * optimize doc-level monitor workflow for index patterns Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> * optimize doc-level monitor execution workflow for datastreams (#1302) * optimize doc-level monitor execution for datastreams Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> * add more tests to address comments Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> * add integTest for multiple datastreams inside a single index pattern * add integTest for multiple datastreams inside a single index pattern Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> --------- Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> * Add jvm aware setting and max num docs settings for batching docs for percolate queries (#1435) * add jvm aware and max docs settings for batching docs for percolate queries Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix stats logging Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add queryfieldnames field in findings mapping Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * optimize to fetch only fields relevant to doc level queries in doc level monitor instead of entire _source for each doc (#1441) * optimize to fetch only fields relevant to doc level queries in doc level monitor Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix test for settings check Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix ktlint Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * optimize sequence number calculation and reduce search requests in doc level monitor execution (#1445) * optimize sequence number calculation and reduce search requests by n where n is number of shards being queried in the executino Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix tests Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * optimize check indices and execute to query only write index of aliases and datastreams during monitor creation Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix test Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add javadoc Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add tests to verify seq_no calculation Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * clean up doc level queries on dry run (#1430) Signed-off-by: Joanne Wang <jowg@amazon.com> * Fix some tests from cherry-picks Signed-off-by: Chase Engelbrecht <engechas@amazon.com> --------- Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Signed-off-by: Chase Engelbrecht <engechas@amazon.com> Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> Signed-off-by: Joanne Wang <jowg@amazon.com> Co-authored-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: Subhobrata Dey <sbcd90@gmail.com> Co-authored-by: Surya Sashank Nistala <snistala@amazon.com> Co-authored-by: Joanne Wang <jowg@amazon.com>

…, #1441 to 2.9 (#1469) * optimize doc-level monitor execution workflow for datastreams (#1302) * optimize doc-level monitor execution for datastreams Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> * add more tests to address comments Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> * add integTest for multiple datastreams inside a single index pattern * add integTest for multiple datastreams inside a single index pattern Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> --------- Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> * Bulk index findings and sequentially invoke auto-correlations (#1355) * Bulk index findings and sequentially invoke auto-correlations Signed-off-by: Megha Goyal <goyamegh@amazon.com> * Bulk index findings in batches of 10000 and make it configurable Signed-off-by: Megha Goyal <goyamegh@amazon.com> * Addressing review comments Signed-off-by: Megha Goyal <goyamegh@amazon.com> * Add integ tests to test bulk index findings Signed-off-by: Megha Goyal <goyamegh@amazon.com> * Fix ktlint formatting Signed-off-by: Megha Goyal <goyamegh@amazon.com> --------- Signed-off-by: Megha Goyal <goyamegh@amazon.com> * fix for MapperException[the [enabled] parameter can't be updated for the object mapping [metadata.source_to_query_index_mapping] (#1432) (#1434) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * Add jvm aware setting and max num docs settings for batching docs for percolate queries (#1435) * add jvm aware and max docs settings for batching docs for percolate queries Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix stats logging Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add queryfieldnames field in findings mapping Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * clean up doc level queries on dry run (#1430) Signed-off-by: Joanne Wang <jowg@amazon.com> * optimize to fetch only fields relevant to doc level queries in doc level monitor instead of entire _source for each doc (#1441) * optimize to fetch only fields relevant to doc level queries in doc level monitor Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix test for settings check Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix ktlint Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * optimize sequence number calculation and reduce search requests in doc level monitor execution (#1445) * optimize sequence number calculation and reduce search requests by n where n is number of shards being queried in the executino Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix tests Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * optimize check indices and execute to query only write index of aliases and datastreams during monitor creation Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix test Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add javadoc Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add tests to verify seq_no calculation Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add distributed locking to jobs in alerting (#1403) Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> * 2.9.1 version bump Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix compilation issues Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * dummy commit Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix findings index schema version tsts Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> Signed-off-by: Megha Goyal <goyamegh@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> Signed-off-by: Joanne Wang <jowg@amazon.com> Co-authored-by: Subhobrata Dey <sbcd90@gmail.com> Co-authored-by: Megha Goyal <56077967+goyamegh@users.noreply.github.com> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: Joanne Wang <jowg@amazon.com>

… percolate queries (opensearch-project#1435) * add jvm aware and max docs settings for batching docs for percolate queries Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix stats logging Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add queryfieldnames field in findings mapping Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

… percolate queries (opensearch-project#1435) * add jvm aware and max docs settings for batching docs for percolate queries Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix stats logging Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add queryfieldnames field in findings mapping Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

* log error messages and clean up monitor when indexing doc level queries or metadata creation fails (#900) * log errors and clean up monitor when indexing doc level queries or metadata creation fails * refactor delete monitor action to re-use delete methods Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> Signed-off-by: Chase Engelbrecht <engechas@amazon.com> * optimize doc-level monitor workflow for index patterns (#1097) Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> Signed-off-by: Chase Engelbrecht <engechas@amazon.com> * optimize doc-level monitor execution workflow for datastreams (#1302) * optimize doc-level monitor execution for datastreams Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> * add more tests to address comments Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> * add integTest for multiple datastreams inside a single index pattern * add integTest for multiple datastreams inside a single index pattern Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> --------- Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> Signed-off-by: Chase Engelbrecht <engechas@amazon.com> * Bulk index findings and sequentially invoke auto-correlations (#1355) * Bulk index findings and sequentially invoke auto-correlations Signed-off-by: Megha Goyal <goyamegh@amazon.com> * Bulk index findings in batches of 10000 and make it configurable Signed-off-by: Megha Goyal <goyamegh@amazon.com> * Addressing review comments Signed-off-by: Megha Goyal <goyamegh@amazon.com> * Add integ tests to test bulk index findings Signed-off-by: Megha Goyal <goyamegh@amazon.com> * Fix ktlint formatting Signed-off-by: Megha Goyal <goyamegh@amazon.com> --------- Signed-off-by: Megha Goyal <goyamegh@amazon.com> Signed-off-by: Chase Engelbrecht <engechas@amazon.com> * Add jvm aware setting and max num docs settings for batching docs for percolate queries (#1435) * add jvm aware and max docs settings for batching docs for percolate queries Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix stats logging Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add queryfieldnames field in findings mapping Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> Signed-off-by: Chase Engelbrecht <engechas@amazon.com> * optimize to fetch only fields relevant to doc level queries in doc level monitor instead of entire _source for each doc (#1441) * optimize to fetch only fields relevant to doc level queries in doc level monitor Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix test for settings check Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix ktlint Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> Signed-off-by: Chase Engelbrecht <engechas@amazon.com> * optimize sequence number calculation and reduce search requests in doc level monitor execution (#1445) * optimize sequence number calculation and reduce search requests by n where n is number of shards being queried in the executino Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix tests Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * optimize check indices and execute to query only write index of aliases and datastreams during monitor creation Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix test Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add javadoc Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add tests to verify seq_no calculation Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> Signed-off-by: Chase Engelbrecht <engechas@amazon.com> * Fix tests Signed-off-by: Chase Engelbrecht <engechas@amazon.com> * Fix BWC tests Signed-off-by: Chase Engelbrecht <engechas@amazon.com> * clean up doc level queries on dry run (#1430) Signed-off-by: Joanne Wang <jowg@amazon.com> Signed-off-by: Chase Engelbrecht <engechas@amazon.com> * Fix import Signed-off-by: Chase Engelbrecht <engechas@amazon.com> * Fix tests Signed-off-by: Chase Engelbrecht <engechas@amazon.com> * Fix BWC version Signed-off-by: Chase Engelbrecht <engechas@amazon.com> * Fix another test Signed-off-by: Chase Engelbrecht <engechas@amazon.com> * Revert order of operations change Signed-off-by: Chase Engelbrecht <engechas@amazon.com> --------- Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> Signed-off-by: Chase Engelbrecht <engechas@amazon.com> Signed-off-by: Megha Goyal <goyamegh@amazon.com> Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> Signed-off-by: Joanne Wang <jowg@amazon.com> Co-authored-by: Surya Sashank Nistala <snistala@amazon.com> Co-authored-by: Subhobrata Dey <sbcd90@gmail.com> Co-authored-by: Megha Goyal <56077967+goyamegh@users.noreply.github.com> Co-authored-by: Joanne Wang <jowg@amazon.com>

add jvm aware and max docs settings for batching docs for percolate q…

586459e

…ueries Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

eirsep requested review from AWSHurneyt, bowenlan-amzn, engechas, getsaurabh02, lezzago, praveensameneni, qreshi, rishabhmaurya and sbcd90 as code owners February 19, 2024 10:19

sbcd90 reviewed Feb 19, 2024

View reviewed changes

alerting/src/main/kotlin/org/opensearch/alerting/DocumentLevelMonitorRunner.kt Show resolved Hide resolved

sbcd90 reviewed Feb 19, 2024

View reviewed changes

alerting/src/main/kotlin/org/opensearch/alerting/AlertingPlugin.kt Show resolved Hide resolved

sbcd90 reviewed Feb 19, 2024

View reviewed changes

alerting/src/main/kotlin/org/opensearch/alerting/DocumentLevelMonitorRunner.kt Outdated Show resolved Hide resolved

fix stats logging

5bf9f72

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

engechas reviewed Feb 19, 2024

View reviewed changes

engechas approved these changes Feb 20, 2024

View reviewed changes

add queryfieldnames field in findings mapping

25d722a

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

eirsep force-pushed the percolate_batching branch from e05f344 to 25d722a Compare February 21, 2024 08:26

sbcd90 approved these changes Feb 21, 2024

View reviewed changes

eirsep merged commit f643454 into opensearch-project:main Feb 21, 2024

eirsep added the backport 2.x label Feb 21, 2024

opensearch-trigger-bot bot added the failed backport label Feb 21, 2024

eirsep removed backport 2.x failed backport labels Feb 22, 2024

opensearch-project deleted a comment from opensearch-trigger-bot bot Feb 22, 2024

eirsep added the backport 2.x label Feb 22, 2024

opensearch-trigger-bot bot added the failed backport label Feb 22, 2024

eirsep mentioned this pull request Mar 13, 2024

[Backport 2.9] Backport #1302, #1355, #1432, #1435, #1445, #1403, #1430, #1441 to 2.9 #1469

Merged

jowg-amazon mentioned this pull request Mar 14, 2024

[Backport to 2.11] Backport #1355, #1435, #1441, #1430, #1445 to 2.11 #1471

Merged

1 task

goyamegh mentioned this pull request Mar 14, 2024

Backport 1122, 1302, 1355, 1435 and 1441 to 2.7 #1472

Closed

1 task

engechas mentioned this pull request Mar 14, 2024

Backport 1097, 1302, 1435, 1441, 1445, 1430 #1470

Merged

1 task

engechas mentioned this pull request Mar 18, 2024

Backports 2.7 #1482

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add jvm aware setting and max num docs settings for batching docs for percolate queries#1435

Add jvm aware setting and max num docs settings for batching docs for percolate queries#1435
eirsep merged 3 commits intoopensearch-project:mainfrom
eirsep:percolate_batching

eirsep commented Feb 19, 2024 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

engechas Feb 19, 2024

Uh oh!

eirsep Feb 19, 2024

Uh oh!

eirsep Feb 20, 2024

Uh oh!

engechas Feb 20, 2024

Uh oh!

eirsep Feb 20, 2024

Uh oh!

opensearch-trigger-bot bot commented Feb 22, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		val queryIndices =
		docs.map { monitorMetadata.sourceToQueryIndexMapping[it.second.indexName + monitor.id] }.distinct()

Conversation

eirsep commented Feb 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Old Latency of percolate query : 5+ minutes leading to cancellation. New latency <1 minute

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

engechas Feb 19, 2024

Choose a reason for hiding this comment

Uh oh!

eirsep Feb 19, 2024

Choose a reason for hiding this comment

Uh oh!

eirsep Feb 20, 2024

Choose a reason for hiding this comment

Uh oh!

engechas Feb 20, 2024

Choose a reason for hiding this comment

Uh oh!

eirsep Feb 20, 2024

Choose a reason for hiding this comment

Uh oh!

opensearch-trigger-bot bot commented Feb 22, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

eirsep commented Feb 19, 2024 •

edited

Loading